Image generation apparatus, image generation system, and image generation method

ABSTRACT

An image processing section  750  draws an image viewed in a line-of-sight direction from a viewpoint position at a certain time. An image storage section  760  holds the image drawn by the image processing section  750  as past frames. A past frame selection section  790  selects, from among the past frames held in the image storage section  760 , a substitute image to be viewed in the line-of-sight direction from a current viewpoint position. An image providing section  770  outputs the substitute image before providing the image viewed in the line-of-sight direction from the current viewpoint position, drawn by the image processing section  750.

TECHNICAL FIELD

The present invention relates to an apparatus, system, and method forgenerating an image.

BACKGROUND ART

A head-mounted display (HMD) connected to a game machine is worn on thehead in such a manner that a user operates a controller etc. andperforms gameplay while viewing a screen displayed on the head-mounteddisplay. When the head-mounted display is worn, the user does not viewany image except a video displayed on the head-mounted display.Therefore, an effect in which immersion feeling in a video world becomeshigher and an entertainment property of a game is more enhanced isexerted. Further, a virtual reality video is displayed on thehead-mounted display and the user who wears the head-mounted displayturns the head. When a virtual space of the entire periphery to overlooka 360-degree direction is displayed, the immersion feeling in a videobecomes higher and even operability of an application of game etc. isfurther improved.

SUMMARY Technical Problems

Thus, a head tracking function is allowed to be held in the head-mounteddisplay and then a viewpoint and a line-of-sight direction are changedin conjunction with a motion of the head of the user to generate thevirtual reality video. In such a case, since latency is generated fromgeneration up to display of the virtual reality video, a deviation maybe generated between a direction of the head of the user to be assumedat the time of generating the video and the direction of the head of theuser at the time of displaying the video on the head-mounted display andthe user may have a feeling as if the user feels sick. In addition, thevirtual reality video generated by a server is received via a networkand is displayed in a client. In such a case, a delay of the network isfurther added and therefore a deviation becomes large between ageneration time and display time of the video.

The present invention has been made in view of the above circumstances.An object of the invention is to provide an image generation apparatusand an image generation method capable of displaying an image inconsideration of latency from generation up to display of the image.

Solution to Problems

In order to solve the problems described above, an image generationapparatus according to a mode of the present invention includes an imagegeneration section that draws an image viewed in a line-of-sightdirection from a viewpoint position at a certain time, a storage sectionthat holds the image drawn by the image generation section as pastframes, a selection section that selects a substitute image to be viewedin the line-of-sight direction from a current viewpoint position fromamong the past frames held in the storage section, and an imageproviding section that outputs the substitute image before providing animage viewed in the line-of-sight direction from the current viewpointposition drawn by the image generation section.

Also another mode of the present invention is an image generationsystem. The image generation system includes a server, and a client thatis connected to the server via a network. The server includes an imagegeneration section that draws an image viewed in a line-of-sightdirection from a viewpoint position at a certain time, a storage sectionthat holds the image drawn by the image generation section as pastframes, a selection section that selects a substitute image to be viewedin the line-of-sight direction from a current viewpoint position fromamong the past frames held in the storage section, and an imageproviding section that transmits the substitute image to the clientbefore providing an image viewed in the line-of-sight direction from thecurrent viewpoint position drawn by the image generation section. Theclient includes a reprojection section that performs processing ofdeviating the substitute image received from the server in accordancewith a motion amount of at least one of the viewpoint position and theline-of-sight direction.

A further mode of the present invention is an image generation method.The method includes an image generation step of drawing an image viewedin a line-of-sight direction from a viewpoint position at a certaintime, a storage step of holding the image drawn in the image generationstep as past frames, a selection step of selecting a substitute image tobe viewed in the line-of-sight direction from a current viewpointposition from among the held past frames, and an image providing step ofoutputting the substitute image before providing an image viewed in theline-of-sight direction from the current viewpoint position drawn in theimage generation step.

Where suitable combinations of the above-described elements or the aboveexpressions of the present invention are converted between differentforms such as a method, a device, a system, a computer program, a datastructure, and a recording medium, they still constitute effectiveembodiments of the present invention.

Advantageous Effect of Invention

According to the present invention, it is possible to display an imagein consideration of latency from generation up to display of the image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an external diagram of a head-mounted display.

FIG. 2 is a functional configuration diagram of the head-mounteddisplay.

FIG. 3 is a configuration diagram of an image generation systemaccording to an embodiment.

FIG. 4 is a functional configuration diagram of an image generationapparatus according to the embodiment.

FIG. 5 is a diagram describing the entire periphery image displayed onthe head-mounted display.

FIG. 6 is a diagram describing a reason that a delay occurs in theentire periphery image displayed on the head-mounted display.

FIGS. 7(a) and 7(b) are diagrams describing a reprojection.

FIG. 8 is a diagram describing an appearance in which a past frame isstored in an image storage section illustrated in FIG. 4.

FIG. 9 is a diagram describing an appearance in which the past frameselected from the image storage section illustrated in FIG. 4 is usedfor drawing processing.

FIG. 10 is a diagram describing past frame selection processing by apast frame selection section illustrated in FIG. 4.

FIG. 11 is a diagram describing past frame discarding processing by apast frame discarding section illustrated in FIG. 4.

FIG. 12 is a flowchart illustrating a procedure of the drawingprocessing using the past frame by the image generation apparatusillustrated in FIG. 4 in the case of a stand-alone system.

FIG. 13 is a flowchart illustrating a procedure of the past frameselection processing illustrated in FIG. 12.

FIG. 14 is a flowchart illustrating a procedure of the past framediscarding processing illustrated in FIG. 12.

FIG. 15 is a flowchart illustrating a procedure of the drawingprocessing using the past frame by the image generation apparatusillustrated in FIG. 4 in the case of a server client system.

DESCRIPTION OF EMBODIMENT

FIG. 1 is an external diagram of a head-mounted display 100. Thehead-mounted display 100 is a display apparatus that is worn on the headof the user in such a manner that the user appreciates a still image, amoving image, or the like displayed on a display and listens to sound,music, or the like output from a headphone.

A gyro sensor, an acceleration sensor, or the like incorporated in orexternally attached to the head-mounted display 100 permits measurementof posture information such as a rotation angle and inclination of thehead and positional information of the user's head wearing thehead-mounted display 100.

On the head-mounted display 100, further, a camera that photographs theeyes of the user may be provided. It is possible to detect a gazedirection, motions of pupils, blinks, and the like of the user by usingthe camera mounted on the head-mounted display 100.

The head-mounted display 100 is an example of a “wearable display.”Here, a method for generating an image displayed on the head-mounteddisplay 100 will be described. In addition to the head-mounted display100 in a narrowly defined sense, the image generation method of thepresent embodiment is applicable to even a case in which glasses, aglass type display, a glass type camera, headphones, a headset(headphones equipped with a microphone), earphones, earrings, anear-hanging camera, a hat, a camera-equipped hat, a hairband, or thelike is worn.

FIG. 2 is a functional configuration diagram of the head-mounted display100.

A control section 10 is a main processor that processes and outputssignals such as image and sensor signals as well as instructions anddata. An input interface 20 receives operating and setting signals fromthe user and supplies the operating and setting signals to the controlsection 10. An output interface 30 receives the image signal from thecontrol section 10 and has the image signal displayed on the display. Abacklight 32 supplies backlight to a liquid crystal display.

A communication control section 40 transmits data input from the controlsection 10 to the outside via a network adapter 42 or an antenna 44 inwired or wireless communication. Also, the communication control section40 receives data from the outside via the network adapter 42 or theantenna 44 in wired or wireless communication and outputs the receiveddata to the control section 10.

A storage section 50 temporarily stores data, parameters, operatingsignals, and the like to be processed by the control section 10.

A posture sensor 64 detects the posture information such as a position,rotation angle, and inclination of the head-mounted display 100. Theposture sensor 64 is implemented by suitably combining a gyro sensor, anacceleration sensor, and an angular acceleration sensor, for example.The motion of backward-and-forward, right-and-left, and up-and-downdirections of the user's head may be detected by using a motion sensorobtained by combining at least one or more of a triaxial geomagneticsensor, a triaxial acceleration sensor, and a triaxial gyro (angularvelocity) sensor.

An external input/output terminal interface 70 is an interface forconnecting peripheral devices such as a universal serial bus (USB)controller. An external memory 72 is an external memory such as a flashmemory.

A clock section 80 sets time information on the basis of the settingsignals from the control section 10 and supplies time data to thecontrol section 10.

The control section 10 can supply images and text data to the outputinterface 30 for display on the display, or supply them to thecommunication control section 40 for transmission to the outside.

FIG. 3 is a configuration diagram illustrating an image generationsystem according to the present embodiment. The head-mounted display 100is connected to a client 200 such as a game machine, a personalcomputer, or a mobile terminal via an interface 300 that connectsperipheral equipment such as wireless communication or a USB. The client200 is further connected to a server 400 via a network 600. The server400 can provide a cloud service. The server 400 is not limited to asingle set and further may be provided in plurality. Further, theplurality of servers 400 may carry out one task by using distributedprocessing.

The server 400 can transmit the virtual reality video to a plurality ofclients 200 via the network 600. A plurality of users wear thehead-mounted display 100 connected to the respective clients 200 andbrowse the virtual reality video from respective viewpoints.

FIG. 4 is a functional configuration diagram illustrating an imagegeneration apparatus 700 according to the present embodiment. The figuredepicts a block diagram aimed at the functions. These functional blocksmay be implemented variously by hardware alone, by software alone, or bycombinations of both.

In the case of a server client system in which the client 200 isconnected to the server 400, the image generation apparatus 700 isimplemented in the server 400. In the case of a stand-alone system inwhich the client 200 is not connected to the server 400 and is used as asingle body, the image generation apparatus 700 is implemented in theclient 200. In the case of the server client system, a portion of afunctional configuration of the image generation apparatus 700 may beimplemented in the server 400 and the remaining functional configurationmay be implemented in the client 200. Even in any case of the serverclient system and the stand-alone system, at least a portion of functionof the image generation apparatus 700 may be implemented in the controlsection 10 of the head-mounted display 100.

A zoom instruction acquisition section 710 acquires zoom magnificationinstructed by the user via the input interface 20 of the head-mounteddisplay 100. The zoom magnification acquired by the zoom instructionacquisition section 710 is provided for a sensitivity adjustment section720 and an image processing section 750.

A position/rotation information acquisition section 730 acquiresinformation regarding a position and rotation of the head of the userwho wears the head-mounted display 100 on the basis of the postureinformation detected by the posture sensor 64 of the head-mounteddisplay 100.

The position/rotation information acquisition section 730 acquires theposition and rotation of the head of the user on the basis ofsensitivity instructed by the sensitivity adjustment section 720. Forexample, when the user turns the neck, a change in an angle of the headof the user is detected by the posture sensor 64. However, thesensitivity adjustment section 720 instructs the position/rotationinformation acquisition section 730 to neglect a change in the detectedangle until the change in the angle exceeds a predetermined value.

Further, the sensitivity adjustment section 720 adjusts the sensitivityof an angle detection of the head on the basis of the zoom magnificationacquired from the zoom instruction acquisition section 710. As the zoommagnification becomes larger, the sensitivity of the angle detection ofthe head is more reduced. When the zooming is performed, a field angleis made small, and therefore an angle detection sensitivity of the headis reduced to thereby suppress vibrations of a display image owing toswinging of the head.

A coordinate conversion section 740 performs a coordinate conversion forgenerating an image to be displayed on the head-mounted display 100 witha tracking function by using the position and rotation of thehead-mounted display 100 acquired by the position/rotation informationacquisition section 730.

The image processing section 750 reads out image data from an imagestorage section 760 and generates an image viewed in a line-of-sightdirection from a viewpoint position of the user who wears thehead-mounted display 100 at the zoom magnification specified by the zoominstruction acquisition section 710 in accordance with the coordinateconversion by the coordinate conversion section 740 to provide the imagefor an image providing section 770. Here, the image data may be a movingimage or still image content prepared in advance or a rendered computergraphics.

The image providing section 770 provides the image data generated by theimage processing section 750 for the head-mounted display 100.

The image processing section 750 stores, as a “past frame,” thegenerated image in the image storage section 760 along with viewpointinformation and a frame number. The viewpoint information includes theviewpoint position and line-of-sight direction at the time of drawingthe frame.

A viewpoint calculation section 780 calculates a viewpoint position andline-of-sight direction after moving the head on the basis of avariation of the position and rotation of the head-mounted display 100acquired from the position/rotation information acquisition section 730.

A past frame selection section 790 compares a new viewpoint positionand/or line-of-sight direction calculated by the viewpoint calculationsection 780 and a viewpoint position and/or line-of-sight direction ofthe past frame stored in the image storage section 760. When adifference between the viewpoint positions and/or an angle differencebetween the line-of-sight directions are present within predeterminedthresholds, the past frame selection section 790 selects the past framefrom the image storage section 760 in accordance with a selection rule.The past frame selection section 790 provides the selected past framefor the image providing section 770.

Before drawing results are generated in the new viewpoint position andline-of-sight direction by using the image processing section 750, theimage providing section 770 provides, as a substitute image, the pastframe selected by the past frame selection section 790 for thehead-mounted display 100.

When a data amount of the past frame stored in the image storage section760 exceeds a predetermined threshold, a past frame discarding section800 deletes the past frame from the image storage section 760 inaccordance with a discard rule. The past frame discarding section 800compares the new viewpoint position and/or line-of-sight directioncalculated by the viewpoint calculation section 780 and the viewpointposition and/or line-of-sight direction of the past frame stored in theimage storage section 760. When the difference between the viewpointpositions and/or the angle difference between the line-of-sightdirections exceed the predetermined thresholds, the past framediscarding section 800 may delete the past frame from the image storagesection 760 in accordance with the discard rule.

FIG. 5 is a diagram describing the entire periphery image 500 displayedon the head-mounted display 100. When the user faces a front left sideto the entire periphery image 500, an image 510 a is displayed in arange of a field angle 150 a in the direction of a head-mounted display100 a. Further, when the user turns the neck and faces a front rightside, an image 510 b is displayed in a range of a field angle 150 b inthe direction of a head-mounted display 100 b.

Even the viewpoint position and line-of-sight direction for viewing theentire periphery image displayed on the head-mounted display 100 arechanged in accordance with the motion of the head as described above,and therefore the immersion feeling can be enhanced in the entireperiphery image.

FIG. 6 is a diagram describing a reason that a delay occurs in theentire periphery image displayed on the head-mounted display 100. Whenthe user turns the neck and faces the front right side, the image 510 bis generated in a range of the field angle 150 b in the direction of thehead-mounted display 100 b and is displayed on the head-mounted display100. In reality, at the time of displaying the image 510 b, the positionand rotation of the head-mounted display 100 b have been already changedso as to be represented in a symbol 150 c. Therefore, an image to beviewed in the range of the field angle 150 c needs to be originallydisplayed on a head-mounted display 100 c. However, an image that isreally generated and displayed is the image to be viewed in the range ofthe field angle 150 b in the direction of the head-mounted display 100 bslightly before the time. Because of a deviation due to the timedifference, an image in a direction that is slightly deviated from thedirection of the image viewed by the user may be displayed on thehead-mounted display 100 and therefore the user may feel a certain typeof “sickness.”

Thus, it takes time to detect the rotation of the head-mounted display100, decide the next drawing range, allow a central processing unit(CPU) to issue a drawing command, allow a graphics processing unit (GPU)to perform rendering, and output a drawn image to the head-mounteddisplay 100. When the drawing is assumed to be performed, for example,at a frame rate of 60 fps (frame/second), even if the CPU operates atsufficiently high speed, a delay occurs for one frame until the rotationof the head-mounted display 100 is detected and the image is output. Thedelay is approximately 16.67 milliseconds under the condition of theframe rate of 60 fps and the time is a sufficient time for a human beingto sense the deviation.

Therefore, in the past, processing that is referred to as “time warp” or“reprojection” has been performed and aimed at making it hard for ahuman being to sense the deviation. Hereinafter, the “reprojection” willbe described and then a method for reducing a drawing delay sense byusing the past frame according to the present embodiment will bedescribed.

FIGS. 7(a) and 7(b) are diagrams describing the reprojection. FIG. 7(a)illustrates a view frustum 910 at the time of viewing a virtual spacefrom a viewpoint 900. In FIG. 7(a), an image 920 is displayed in adisplay range regulated by the view frustum 910. However, in the case inwhich the user turns the head to the right and moves the viewpoint 900to the right, when the generated image 920 is directly drawn in thedisplay range regulated by the view frustum 910, the user senses thedeviation as illustrated in FIG. 6.

When the head is moved to the right, the image before the eyes ought toflow from the right to the left, to put it more precisely, flow to thelower left. For that purpose, as illustrated in FIG. 7(b), the generatedimage 920 is displayed in a position deviated at the lower left than aposition to be originally displayed. Then, a correct image is late drawnin the original display range regulated by the view frustum 910. Theprocessing is referred to as the reprojection. Through the reprojection,uncomfortable feeling sensed by a human being can be suppressed becausethe image does not instantaneously follow the motion of the head.

However, when the drawing is performed in the head-mounted display 100particularly by using the server client system, a communication time isfurther added to a time required for drawing processing and the delaybecomes larger. Therefore, it may be difficult to sufficiently reducethe uncomfortable feeling by using only delay measures according to thereprojection. The generated image 920 is displayed in a positiondeviated from a position to be originally displayed and then the correctimage is drawn in the original display range regulated by the viewfrustum 910. Further, the drawing of the correct image is delayed inconsequence of communication traffic. To solve the above problem, in thepresent embodiment, usage of the past frame permits the delay sensed bythe user to be reduced.

The drawing processing using the past frame will be schematicallydescribed with reference to FIGS. 8 and 9. FIG. 8 is a diagramdescribing an appearance in which the past frame is stored in the imagestorage section 760. Images 920 a and 920 b are drawn in the displayrange regulated by the view frustums 910 a and 910 b at the time ofviewing the virtual space when moving viewpoints 900 a and 900 b. Imagesfor N pieces of frames (N is an arbitrary natural number) in the pastare stored in the image storage section 760.

FIG. 9 is a diagram describing an appearance in which the past frameselected from the image storage section 760 is used for the drawingprocessing. A viewpoint 900 c is assumed to move in the left direction.It takes time to obtain a drawing result at a new viewpoint. For thatpurpose, the frame 920 a that is drawn in the past at a viewpointnearest to the viewpoint 900 c is selected from among frames of theimages stored in the image storage section 760. The selected frame 920 ais directly used, or the selected frame 920 a that is subjected toreprojection processing if necessary is displayed as a substitute image920 c in the display range regulated by a view frustum 910 c at the timeof viewing the virtual space from the viewpoint 900 c.

Then, an image 920 d to be originally drawn is generated and displayedin the display range of the view frustum 910 c at the time of viewingthe virtual space from a viewpoint 900 d in a movement destination. Thepast frame 920 a is displayed as the substitute image 920 c until theimage 920 d that is a final drawing result at the viewpoint 900 d isdisplayed, and therefore the delay sensed by the user is reduced. Theimage 920 d that is the final drawing result at the viewpoint 900 d isstored as a new past frame in the image storage section 760.

FIG. 10 is a diagram describing selection processing of the past frameby the past frame selection section 790. Here, a case in which a rulesuch that “a past frame that is drawn at a viewpoint nearest to acurrent viewpoint is selected” is adopted as the selection rule will bedescribed.

To put it briefly, a frame FrA drawn at a first viewpoint VpA and aframe FrB drawn at a second viewpoint VpB are assumed to be stored asthe past frame in the image storage section 760. The past frameselection section 790 selects the substitute image viewed from a currentviewpoint VpC from among the past frames stored in the image storagesection 760. When the current viewpoint VpC is compared with the firstviewpoint VpA and the second viewpoint VpB, the second viewpoint VpB isnear to the current viewpoint VpC. In the result, the past frameselection section 790 selects the frame FrB drawn at the secondviewpoint VpB nearest to the current viewpoint VpC as the substituteimage in the current viewpoint VpC.

FIG. 11 is a diagram describing discarding processing of the past frameby the past frame discarding section 800. Here, a case in which a rulesuch that “the past frame that is drawn at a viewpoint most distant fromthe current viewpoint is discarded” is adopted as the discard rule willbe described.

To put it briefly, the frame FrA drawn at the first viewpoint VpA andthe frame FrB drawn at the second viewpoint VpB are assumed to be storedas the past frame in the image storage section 760. When the currentviewpoint VpC is compared with the first viewpoint VpA and the secondviewpoint VpB, the first viewpoint VpA is distant from the currentviewpoint VpC. In the result, the past frame discarding section 800deletes the frame FrA that is drawn at the first viewpoint VpA mostdistant from the current viewpoint VpC from the image storage section760.

FIG. 12 is a flowchart illustrating a procedure of the drawingprocessing using the past frame by the image generation apparatus 700 inthe case of the stand-alone system.

The image generation apparatus 700 acquires a parameter determinedduring initial setting, such as a display resolution and camera fieldangle of the head-mounted display 100 and stores the parameter in astorage section as a parameter for viewpoint determination (S10).

The position/rotation information acquisition section 730 of the imagegeneration apparatus 700 acquires a real-time measurement valueindicating the motion etc. of the head of the user who wears thehead-mounted display 100 and stores the real-time measurement value inthe storage section as the parameter for viewpoint determination (S12).

The viewpoint calculation section 780 of the image generation apparatus700 calculates the viewpoint position and line-of-sight direction forperforming the drawing on the basis of the stored parameter forviewpoint determination (S14).

The past frame selection section 790 compares the viewpoint position andline-of-sight direction of the past frame stored in the image storagesection 760 and the viewpoint position and line-of-sight directioncalculated by the viewpoint calculation section 780 and determineswhether a difference between the viewpoint positions and an angledifference between the line-of-sight directions are present withinpredetermined thresholds (S16).

If the difference between the viewpoint positions and the angledifference between the line-of-sight directions are present within thepredetermined thresholds (Y at S16), the past frame selection section790 performs past frame selection processing (S18). Details of the pastframe selection processing will be described below. The past frameselection processing is performed, and then the process proceeds to stepS20.

If the difference between the viewpoint positions and the angledifference between the line-of-sight directions exceed the predeterminedthresholds (N at S16), the past frame selection processing of step S18is not performed and the process proceeds to step S20.

The image processing section 750 draws an image viewed in theline-of-sight direction from a current viewpoint position and providesthe drawn image for the image providing section 770 (S20).

The past frame discarding section 800 determines whether the data amountof the past frames stored in the image storage section 760 exceeds apredetermined threshold (S22). The predetermined threshold is set underthe control of the data amount capable of being held in the imagegeneration apparatus 700 in consideration of a storage capacity of theimage generation apparatus 700.

If the data amount of the past frames stored in the image storagesection 760 exceeds the predetermined threshold (Y at S22), the pastframe discarding section 800 performs past frame discarding processing(S24). Details of the past frame discarding processing will be describedbelow. The past frame discarding processing is performed, and then theprocess proceeds to step S26.

If the data amount of the past frames stored in the image storagesection 760 is present within the predetermined threshold (N at S22),the past frame discarding processing of step S24 is not performed andthe process proceeds to step S26.

The image processing section 750 stores, as the past frame, the drawingresult of the image viewed in the line-of-sight direction from thecurrent viewpoint position in the image storage section 760 along withthe viewpoint and the frame number (S26).

The image providing section 770 outputs the drawing result of the imageviewed in the line-of-sight direction from the current viewpointposition to the head-mounted display 100 and performs the reprojectionprocessing if necessary (S28).

If the head of the user is moved (Y at S30), the process returns to stepS12 and subsequent processes are repeated. If the head of the user isnot moved (N at S30), the drawing processing using the past frame endsand the process returns to normal drawing processing.

FIG. 13 is a flowchart illustrating a procedure of the past frameselection processing of step S18 illustrated in FIG. 12.

The past frame selection section 790 selects the past frame from theimage storage section 760 on the basis of the selection rule (S40).

Examples of the selection rule include the following.

-   -   Select a newest frame among from the stored past frames    -   Select the past frame drawn at a viewpoint nearest to the        current viewpoint on the basis of the difference between the        viewpoint positions and the angle difference between the        line-of-sight directions    -   Perform matching of the immediately drawn frame and the past        frame and select the past frame that most closely approximates        the immediately drawn frame    -   Select the past frame having a highest selection frequency

The past frame selection section 790 outputs the selected past frame asthe substitute image to the head-mounted display 100 and performs thereprojection processing if necessary (S42).

FIG. 14 is a flowchart illustrating a procedure of the past framediscarding processing of step S24 illustrated in FIG. 12.

The past frame discarding section 800 selects the past frame to bediscarded from the image storage section 760 on the basis of the discardrule (S50).

Examples of the discard rule include the following.

-   -   Discard an oldest frame from among the stored past frames    -   Discard the past frame drawn at a viewpoint most distant from        the current viewpoint on the basis of the difference between the        viewpoint positions and the angle difference between the        line-of-sight directions    -   Perform matching of the immediately drawn frame and the past        frame and discard the past frame that does not most closely        approximate the immediately drawn frame    -   Discard the past frame having a lowest selection frequency

The past frame discarding section 800 deletes the selected past framefrom the image storage section 760 (S52).

In the above-described flowchart, a case of the stand-alone system inwhich the client 200 is not connected to the server 400 and is used as asingle body is described. Next, a case of the server client system inwhich the client 200 is connected to the server 400 will be described.

FIG. 15 is a flowchart illustrating a procedure of the drawingprocessing using the past frame by the image generation apparatus 700 inthe case of the server client system.

The client 200 acquires a parameter determined during initial setting,such as the display resolution and camera field angle of thehead-mounted display 100 and transmits the parameter to the server 400.The server 400 stores the received parameter in the storage section asthe parameter for viewpoint determination (S60).

The client 200 acquires the real-time measurement value of the motionetc. of the head of the user who wears the head-mounted display 100 andtransmits the real-time measurement value to the server 400. Then, theposition/rotation information acquisition section 730 of the server 400stores the received real-time measurement value in the storage sectionas the parameter for viewpoint determination (S62).

The viewpoint calculation section 780 of the server 400 calculates theviewpoint position and line-of-sight direction for performing drawingfrom the stored parameter for viewpoint determination (S64).

The past frame selection section 790 of the server 400 compares theviewpoint position and line-of-sight direction of the past frame storedin the image storage section 760 and the viewpoint position andline-of-sight direction calculated by the viewpoint calculation section780 and determines whether a difference between the viewpoint positionsand an angle difference between the line-of-sight directions are presentwithin predetermined thresholds (S66).

If the difference between the viewpoint positions and the angledifference between the line-of-sight directions are present within thepredetermined thresholds (Y at S66), the past frame selection section790 performs the past frame selection processing (S68). Details of thepast frame selection processing will be described below. The past frameselection processing is performed, and then the process proceeds to stepS70.

If the difference between the viewpoint positions and the angledifference between the line-of-sight directions exceed the predeterminedthresholds (N at S66), the past frame selection processing of step S68is not performed and the process proceeds to step S70.

The image processing section 750 of the server 400 draws an image viewedin the line-of-sight direction from the current viewpoint position andprovides the drawn image for the image providing section 770 (S70).

The past frame discarding section 800 of the server 400 determineswhether the data amount of the past frames stored in the image storagesection 760 exceeds a predetermined threshold (S72). The predeterminedthreshold is set on the basis of the data amount capable of being heldin the server 400 in consideration of a high-capacity storage apparatusof the server 400. In the case of the cloud service, the storagecapacity is sufficiently large, and therefore a large number of pastframes that are transmitted to the plurality of clients 200 can be held.

If the data amount of the past frames stored in the image storagesection 760 exceeds the predetermined threshold (Y at S72), the pastframe discarding section 800 performs the past frame discardingprocessing (S74). Details of the past frame discarding processing willbe described below. The past frame discarding processing is performed,and then the process proceeds to step S76.

If the data amount of the past frames stored in the image storagesection 760 is present within the predetermined threshold (N at S72),the past frame discarding processing of step S74 is not performed andthe process proceeds to step S76.

The image processing section 750 of the server 400 stores, as the pastframe, the drawing result of the image viewed in the line-of-sightdirection from the current viewpoint position in the image storagesection 760 along with the viewpoint and the frame number (S76).

The image providing section 770 of the server 400 transmits the drawingresult of the image viewed in the line-of-sight direction from thecurrent viewpoint position to the client 200. The client 200 outputs thereceived drawing result to the head-mounted display 100 and performs thereprojection processing if necessary (S78).

If the head of the user is moved (Y at S80), the process returns to stepS62 and subsequent processes are repeated. If the head of the user isnot moved (N at S80), the drawing processing using the past frame endsand the process returns to the normal drawing processing.

Since the past frame selection processing of step S68 is not sodifferent from processing in the case of the stand-alone systemillustrated in FIG. 13, figures are omitted and only procedures will bedescribed. The past frame selection section 790 of the server 400selects the past frame from the image storage section 760 on the basisof the selection rule and transmits the past frame to the client 200.The client 200 outputs the received past frame as the substitute imageto the head-mounted display 100 and performs the reprojection processingif necessary.

Since the past frame selection processing of step S74 is not sodifferent from the processing in the case of the stand-alone systemillustrated in FIG. 14, figures are omitted and only procedures will bedescribed. The past frame discarding section 800 of the server 400selects the past frame to be discarded from the image storage section760 on the basis of the discard rule. The past frame discarding section800 deletes the selected past frame from the image storage section 760.

In the descriptions, a case in which the server 400 transmits thedrawing result to a single client 200 is described; further, the server400 may transmit the drawing result to the plurality of clients 200. Forexample, when a game application is performed by using the plurality ofclients 200, the plurality of users view the same virtual space fromdifferent viewpoints while wearing the head-mounted display 100. In sucha case, a past frame of an image drawn for one client 200 is stored inthe server 400. Then, the past frame may be reused as the substituteimage in the case in which another client 200 views the virtual spacefrom a similar viewpoint.

As described above, according to the image generation apparatus 700 ofthe present embodiment, the viewpoint position and the line-of-sightdirection are changed in accordance with movements of the head of theuser who wears the head-mounted display 100. In such a case, the pastframe suitable for a new viewpoint position and line-of-sight directionis reused from among the past frames of the images drawn in the past andis drawn as the substitute image. In the result, even if the delay iscaused by the drawing processing or communication, the substitute imageis displayed. Therefore, it is hard for the user to sense the delay andthe sickness can be prevented.

Hereinafter, several use cases will be described. A first use case is amode in which a device under the user's hand performs drawing by usingthe past frame without connecting the device to a network. When thedevice under the user's hand has a sufficient storage capacity, thedrawing result in the past is stored and reused to thereby output thedrawing in which the delay required for the drawing processing ishidden.

A second use case is a mode in which the drawing is performed in a cloudserver and the drawing result is transmitted to the client. The cloudserver has a storage apparatus with higher capacity than that of thedevice under the user's hand, and therefore can hold and reuse manydrawing results in the past as compared with the device under the user'shand. Further, in the cloud service, when the plurality of clients 200view the same virtual space, a drawing result of one client can be usedby another client.

A third use case is a mode in which reproduction of a moving imagecontent of the entire circumference in 360 degrees is performed by thecloud server and only an area viewed in the line-of-sight direction ofthe user is segmented to transmit the area to the client. For example,hardware with high performance is needed for the reproduction of acontent with high resolution such as 4 K resolution and 8 K resolution.An environment is needed for providing hardware with higher performancethan that of a terminal under the user's hand as in the cloud service.

To transmit all contents with the high resolution such as 4 K resolutionand 8 K resolution to the clients, an ultra-wide-band networkcommunication is needed. For that purpose, only an area viewed in theline-of-sight direction of the user is segmented to transmit the area tothe client. This process permits the necessary network band to bereduced while using a capability of the cloud service. However, only aportion of an area of the reproduced moving images is transmitted to theclient, and therefore when the viewpoint moves, the reprojectionprocessing is needed. To solve the problem, frames of the imagesegmented from a 360-degree moving image are stored in the cloud server.Further, the past frame suitable for a viewpoint at the time when theclient reproduces a 360-degree moving image content is selected andreused as the substitute image to thereby hide the delay.

The present invention has been described on the basis of theembodiments. The embodiments are illustrative, and they can berecognized by those skilled in the art that various modifications arepossible in regard to suitable combinations of the components and theprocesses of the embodiments and that also such modifications fallwithin the scope of the present invention. Such modifications aredescribed.

REFERENCE SIGNS LIST

10 Control section, 20 Input interface, 30 Output interface, 32Backlight, 40 Communication control section, 42 Network adapter, 44Antenna, 50 Storage section, 64 Posture sensor, 70 External input/outputterminal interface, 72 External memory, 80 Clock section, 100Head-mounted display, 200 Client, 300 Interface, 400 Server, 500 Entireperiphery image, 600 Network, 700 Image generation apparatus, 710 Zoominstruction acquisition section, 720 Sensitivity adjustment section, 730Position/rotation information acquisition section, 740 Coordinateconversion section, 750 Image processing section, 760 Image storagesection, 770 Image providing section, 780 Viewpoint calculation section,790 Past frame selection section, 800 Past frame discarding section.

INDUSTRIAL APPLICABILITY

The present invention can be applied to a technique for generating animage.

The invention claimed is:
 1. An image generation apparatus for compensating for drawing delays, the image generating apparatus comprising: an image generation section that draws a display image viewed in a line-of-sight direction from a viewpoint position at a certain time, wherein the display image displays a virtual three-dimensional space from the viewpoint position on a display of a non-transmissive head mounted display (HMD), a storage section that holds already displayed display images as past frames for a predetermined period of time; a selection section that selects a substitute image to be viewed in the line-of-sight direction from a current viewpoint position from among the past frames held in the storage section; and an image providing section that displays the substitute image on the display, to compensate for drawing delays, before displaying a new image viewed in the line-of-sight direction from the current viewpoint position drawn by the image generation section, wherein the display image, the past frames, and the substitute image are all full frame images covering an entirety of the display.
 2. The image generation apparatus according to claim 1, wherein the selection section compares at least one of the viewpoint position and line-of-sight direction of the past frame held in the storage section and at least one of the current viewpoint position and the line-of-sight direction and selects, as the substitute image, a past frame in which at least one of a difference between the viewpoint positions and an angle difference between the line-of-sight directions is smallest.
 3. The image generation apparatus according to claim 2, wherein when at least one of the difference between the viewpoint positions and the angle difference between the line-of-sight directions is present within a predetermined threshold, the selection section selects the substitute image from among the past frames held in the storage section, and when at least one of the difference between the viewpoint positions and the angle difference between the line-of-sight directions exceeds the predetermined threshold, the selection section does not select the substitute image and the image providing section does not output the substitute image and provides the display image viewed in the line-of-sight direction from the current viewpoint position drawn by the image generation section.
 4. The image generation apparatus according to claim 1, further comprising: a discarding section that compares, in the past frames held in the storage section, at least one of the viewpoint position and line-of-sight direction of the past frame held in the storage section and at least one of the current viewpoint position and the line-of-sight direction and deletes, from the storage section, a past frame in which at least one of the difference between the viewpoint positions and the angle difference between the line-of-sight directions is largest.
 5. An image generation system comprising: a server; and a client that is connected to the server via a network, wherein the server includes an image generation section that draws a display image viewed in a line-of-sight direction from a viewpoint position at a certain time, a storage section that holds a plurality of already displayed display images as past frames, a selection section that selects a substitute image to be viewed in the line-of-sight direction from a current viewpoint position from among the past frames held in the storage section, and an image providing section that transmits the substitute image to the client before providing a new display image viewed in the line-of-sight direction from the current viewpoint position drawn by the image generation section, wherein the display image displays a virtual three-dimensional space from the viewpoint position, and wherein the display image, the past frames, and the substitute image are all full frame images covering an entirety of a display of a non-transmissive head mounted display (HMD), and the client includes a reprojection section that performs processing of deviating the substitute image received from the server in accordance with a motion amount of at least one of the viewpoint position and the line-of-sight direction and displaying the deviated image on the display.
 6. An image generation method comprising: drawing a display image viewed in a line-of-sight direction from a viewpoint position at a certain time, wherein the display image displays a virtual three-dimensional space from the viewpoint position on a display of a non-transmissive head mounted display (HMD); storing already displayed display images as past frames for a predetermined period of time; selecting a substitute image to be viewed in the line-of-sight direction from a current viewpoint position from among the past frames; and outputting the substitute image to the display before providing a new image viewed in the line-of-sight direction from the current viewpoint position drawn to the display, wherein the display image, the past frames, and the substitute image are all full frame images covering an entirety of the display. 